Apparatus and method for simulating an analytic value chain

ABSTRACT

A computer-implemented simulator models the entire analytic value chain so that data generation, model fitting and strategy optimization are an integral part of the simulation. Data collection efforts, data mining algorithms, predictive modeling technologies and strategy development methodologies define the analytic value chain of a business operation: data→models→strategies→profit. Inputs to the simulator include consumer data and potential actions to be taken regarding a consumer or account. The invention maps what is known about a consumer or an account and the potential actions that the business can take on that consumer or account to potential future financial performance. After iteratively performing simulations using varying inputs, modeling the effect of the innovation on a profit model, the simulator outputs a prediction of the commercial value of an analytic innovation.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. provisional patent applicationSer. No. 60/706,936, filed Aug. 9, 2005, the entirety of which isincorporated herein by this reference thereto.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Generally, the invention relates to automated decision-making andoptimizing automated decision-making processes. More particularly, theinvention relates to data processing systems and methods for simulatingan analytic value chain.

2. Background Information

Businesses must make a multitude of decisions every day, both large andsmall. These decisions may involve determining what price to charge aparticular customer, whether to grant a loan or an insurance policy, howto route air traffic or whether or not to issue a prescription to aparticular patient. Particularly in financial services industries,entities have traditionally employed large numbers of low- and mid-levelknowledge workers to make many of these decisions, a practice whichoften entailed high operation and opportunity costs to reach decisions.Additionally, traditional decision-making processes can be slow andcumbersome. For example, using traditional methods of mortgageunderwriting, obtaining a loan approval often required several months.The human factor in decision-making can also result in imprecise,inconsistent decisions. Seeking to improve such factors indecision-making as cost, speed, consistency, precision and agility,businesses are turning more and more to automated decision-makingtechnologies.

Using these technologies it becomes possible to build automated systemsthat sense data, apply codified knowledge or logic to the data, and makedecisions with little or no human intervention. Additionally, theInternet has made automated decision-making more feasible. More and moreindividual financial data is obtainable over the Internet in real-time.For example, an individual's FICO (FAIR ISAAC CORPORATION, MinneapolisMinn.) score, which summarizes the consumer's credit relationships andpayment history into one number, is available in a second or two.Consumers easily apply for loans online. Automated decision-making canhelp businesses generate decisions that are more consistent than thosemade by people and can help managers move quickly from insight todecision to action.

Since the early days of scoring and automated decision making a, therehas been a quest to improve data, models, and strategies, with the hopeof improving decision yield, and thereby improving the profit pictureand competitive capacity of a business operation. However, there arecosts and risks associated with introducing changes such as analyticinnovations to a current operation. Even limited field tests can beexpensive to administer and businesses usually desire ROI (return oninvestment) estimates for proposed analytic innovations beforeproceeding to field testing.

SUMMARY OF THE INVENTION

A computer-implemented simulator models the entire analytic value chainso that data generation, model fitting and strategy optimization are anintegral part of the simulation. Data collection efforts, data miningalgorithms, predictive modeling technologies and strategy developmentmethodologies define the analytic value chain of a business operation:data→models→strategies→profit, as described in commonly-assigned U.S.patent application Ser. No. 10/697,907, Method and apparatus forcreating and evaluating strategies. Inputs to the simulator includeconsumer data and potential actions to be taken regarding a consumer oraccount. The invention maps what is known about a consumer or an accountand the potential actions that the business can take on that consumer oraccount to potential future financial performance. After iterativelyperforming simulations using varying inputs, modeling the effect of theinnovation on a profit model, the simulator outputs a prediction of thecommercial value of an analytic innovation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 provides a diagram of a machine in the exemplary form of acomputer system within which a set of instructions, for causing themachine to perform any one of the methodologies discussed herein below,may be executed;

FIG. 2 is a block diagram of a method for quantifying a relationshipbetween approximation quality and profit;

FIG. 3 provides a block diagram of a software implemented engine forsimulating an analytic value chair;

FIG. 4 provides a schematic diagram of an apparatus for simulating ananalytic value chain;

FIG. 5 is a block diagram of a method for estimating value of atransaction risk score for credit card line management;

FIG. 6 shows a graph of uncertain profit distributions resulting fromtwo different approaches to determining credit risk;

FIG. 7 shows a distribution of opportunity from an approach todetermining credit risk that includes a transaction score;

FIG. 8 provides a block diagram of a method for improving accuracy of ascreen for predicting good/bad status of loan applicants based on rejectinference;

FIG. 9 provides a chart comparing score weight patterns for a positedprofit model, a mildly irrational, and a very irrational screen;

FIG. 10 shows a chart providing smoothed histogram of mean accountprofit for a first scenario;

FIG. 11 provides a diagram of a stochastic learning strategy;

FIG. 12 is a diagram showing a behavioral result from following aposited learning strategy;

FIG. 13 provides a block diagram of a method for simulating futureoutcomes of a credit line increase strategy;

FIGS. 14A and B provide block diagrams of stages I and II, respectively,of a method for estimating the benefit of an improvement to a componentmodel; and

FIG. 15 provides a diagram comparing profit from over time fromfollowing two learning strategies, respectively.

DETAILED DESCRIPTION

A computer-implemented simulator models the entire analytic value chainso that data generation, model fitting and strategy optimization are anintegral part of the simulation. Data collection efforts, data miningalgorithms, predictive modeling technologies and strategy developmentmethodologies define the analytic value chain of a business operation:data→models→strategies→profit. Inputs to the simulator include consumerdata and potential actions to be taken regarding a consumer or account.The invention maps what is known about a consumer or an account and thepotential actions that the business can take on that consumer or accountto potential future financial performance. After iteratively performingsimulations using varying inputs, modeling the effect of the innovationon a profit model, the simulator outputs a prediction of the commercialvalue of an analytic innovation.

The notion of the analytic value chain captures the intuition thatimproved data sources allow for more powerful scoring models to bebuilt. Better scores, whether they result from better data or frombetter model estimation strategies enable the development of improveddecision strategies. Superior strategies lead to higher profit. Analyticinnovations offer opportunities to strengthen the analytic value chainand to reap higher profit.

Analytic value chain simulation (AVACS) attempts to rationalize andquantify these intuitive ideas, using a decision-theoretic framework.What is the importance of this? There are costs and risks associatedwith implementing any changes to the current operation. Even limitedfield tests can be expensive to administer. Businesses desire ROI(return on investment) estimates for proposed analytic innovations,before eventually proceeding to field testing. AVACS generates estimatesfor the return.

Several embodiments of the invention are described herein, The severalembodiments all share a common objective and overarching methodology:they are tools that enable the user to learn more about the relationshipbetween observed consumer behavior and potential actions applied to theconsumer on the one hand and future profit on the other, AVACS (analyticvalue chain simulation) posits a known formula to mimic the truerelationship. It carefully distinguishes this “true” relationship froman “estimated” relationship. This framework allows investigation intohow an analytic innovation may eventually lead to an improvedapproximation to the known true relationship, or conversely, how theabsence of an analytic innovation may result in a loss of approximationquality. Unlike statistical measures of fit quality, AVACS goes a stepfurther in that it evaluates the commercial value of an analyticinnovation. It does this by linking improvements in the approximationquality to improvements in the decisions or actions, and, finally, toimprovements in profit.

Herein below, the general principles and features of the invention aredescribed. Afterward, several exemplary implementations of the inventionare described:

-   -   predicting the value of transaction data for credit line        management;    -   predicting the value of reject inference for account        origination; and    -   predicting the value of an experimental design for faster        learning in closed-loop adaptive control.

FIG. 1 shows a diagrammatic representation of a machine in the exemplaryform of a computer system 100 within which a set of instructions, forcausing the machine to perform any one of the methodologies discussedherein below, may be executed. In alternative embodiments, the machinemay comprise a network router, a network switch, a network bridge,personal digital assistant (PDA), a cellular telephone, a web applianceor any machine capable of executing a sequence of instructions thatspecify actions to be taken by that machine.

The computer system 100 includes a processor 102, a main memory 104 anda static memory 106, which communicate with each other via a bus 108.The computer system 100 may further include a display unit 110, forexample, a liquid crystal display (LCD) or a cathode ray tube (CRT). Thecomputer system 100 also includes an alphanumeric input device 112, forexample, a keyboard; a cursor control device 114, for example, a mouse;a disk drive unit 116, a signal generation device 118, for example, aspeaker, and a network interface device 120.

The disk drive unit 116 includes a machine-readable medium 124 on whichis stored a set of executable instructions, i.e. software, 126 embodyingany one, or all, of the methodologies described herein below. Thesoftware 126 is also shown to reside, completely or at least partially,within the main memory 104 and/or within the processor 102. The software126 may further be transmitted or received over a network 128 by meansof a network interface device 120.

In contrast to the system 100 discussed above, a different embodiment ofthe invention uses logic circuitry instead of computer-executedinstructions to implement processing entities. Depending upon theparticular requirements of the application in the areas of speed,expense, tooling costs, and the like, this logic may be implemented byconstructing an application-specific integrated circuit (ASIC) havingthousands of tiny integrated transistors. Such an ASIC may beimplemented with CMOS (complimentary metal oxide semiconductor), TTL(transistor-transistor logic), VLSI (very large systems integration), oranother suitable construction. Other alternatives include a digitalsignal processing chip (DSP), discrete circuitry (such as resistors,capacitors, diodes, inductors, and transistors), field programmable gatearray (FPG3A), programmable logic array (PLA), programmable logic device(PLD), and the like.

It is to be understood that embodiments of this invention may be used asor to support software programs executed upon some form or processingcore (such as the CPU of a computer) or otherwise implemented orrealized upon or within a machine or computer readable medium. Amachine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine, e.g. acomputer. For example, a machine readable medium includes read-onlymemory (ROM); random access memory (RAM); magnetic disk storage media;optical storage media; flash memory devices; electrical, optical,acoustical or other form of propagated signals, for example, carrierwaves, infrared signals, digital signals, etc.; or any other type ofmedia suitable for storing or transmitting information.

Overview Description of AVACS

Posited Profit Model

The cornerstone of the simulation is to posit a model for expectedprofit. It maps what is known about a consumer or an account, and thepotential actions that the business can take on that consumer oraccount, to potential future financial performance. While thedescription herein has been limited to a discussion of profit for thesake of simplicity, the description is intended only to be illustrative.In fact, businesses are almost always interested in multiple, competingperformance objectives, such as profit, growth, and loss. The simulationtool presented herein may be generalized to multi-dimensionalperformance measures. A model for expected future profit from an accountor consumer may have the form:Eprofit=f (X,A;β)  (Eq. 1)where:

-   -   X: Data available about the consumer at time of decision    -   A: Potential actions applicable to the consumer    -   β: Model parameters.

The data X can include anything that provides insight into the consumer,including internal and external data sources:

-   -   application and survey information;    -   transaction patterns and scores; and    -   actions that were previously applied to the consumer or account.

The potential actions A are members of a fixed set of actions pertainingto a particular decision area, for example:

-   -   Accept/Reject for account origination; and/or    -   discrete levels of Line Increase Amount for credit card line        management. The potential actions are under the control of the        decision maker.

The structural form of the profit function and the values of the modelparameters are informed by data and domain knowledge. Preferably, theposited profit model reflects the best available knowledge about thedependency of profit on consumer or account information and actions. Forexample, if there is evidence that profit depends on a variable x, thensuch dependency should be reflected in the model. In one embodiment ofthe invention, it is assumed that this model represents the truerelationship. In another embodiment, described herein below, thisassumption is relaxed.

Approximation Quality and Profit

A great number of analytic innovations serve the pursuit of learningmore about the true relationship f and to approximate it as closely aspossible—among them transaction-based scores, reject inferencetechniques, and experimental design. The commercial value, oropportunity, of such an innovation can be linked to its success atimproving the estimate of f, compared to the approximation qualitywithout the innovation. As shown in FIG. 2, a method 200 for quantifyingthis relationship between approximation quality and profit involves atleast the following steps:

-   -   developing estimates with and without the innovation 202;    -   using the estimates to develop estimated optimal strategies 204;    -   computing the profits arising from these strategies 206; and    -   computing the profit difference, which is defined as the        opportunity of the analytic innovation 208.        Strategy Optimization

Within the context of the invention, a decision strategy is a mappingfrom the consumer or account information Lo the space of actions. Adecision strategy may encompass a set of rules or logic that a businessoperation uses to determine what action to take in a particularcircumstance to achieve a desired result. For example, a collectionsstrategy could include rules that indicate which delinquent customers tocontact, when, how, by which aligns and in which order, in order tomaximize return. The business seeks to uncover the best strategy, givenbusiness objectives and constraints. The present description is limitedto the problem of unconstrained profit maximization. However, thesimulation framework described here generalizes to optimization problemswith multiple objectives and constraints. The optimal strategy is givenby:A*(X)=arg max_({A∈ActionSet})f (X,A;β)  (Eq. 2).

In the real world, f is not known perfectly well, so the optimalstrategy is also not perfectly well known. But we do have estimates forit, so we can determine estimated optimal strategies. Let f⁺(X,A,β⁺) andf⁻(X,A,β⁻) denote our estimates for f with and without the innovation,respectively. The estimated functions can differ in their data sources,their structure, and their parameter values. How we arrive at theseestimates is application specific and will be discussed below in thesections devoted to the various implementations. The estimated optimalstrategies with and without the innovation are, respectively:A ⁺(X)=arg max_(AeActionSet)f⁺(X,A;β ⁺)  (Eq. 3).A ⁻(X)=arg max_(AeActionSet)f⁻(X,A;β ⁻)  (Eq. 4).

Note that because f⁺≠f⁻, it may happen that A⁺≠A⁻.

Expected Profit and Opportunity

By virtue of the posited relationship, we calculate expected profit fromthe two strategies:Eprofit⁺(X)=f (X,A ⁺;β)  (Eq. 5)Eprofit⁻(X)=f (X,A ⁻;β)  (Eq. 6).

If A⁺≠A⁻, this tends to lead to profit differences. The expected totaland mean portfolio profit for a portfolio of size N of individuals oraccounts with characteristics X_(i); i=1, . . . , N is:

$\begin{matrix}{{{Eprofit}_{total}^{+} = {\sum\limits_{i = 1}^{N}{{Eprofit}^{+}\left( X_{i} \right)}}}{Eprofit}_{mean}^{+} = {\frac{1}{N}{{Eprofit}_{total}^{+}.}}} & \left( {{Eq}.\mspace{14mu} 6} \right)\end{matrix}$

(Analogous for Eprofit⁻). The total and mean innovation opportunity is:Oppt_(total)=Eprofit_(total) ⁺−Eprofit_(total) ⁻Oppt_(mean)=Eprofit_(mean) ⁺−Eprofit_(mean) ⁻  (Eq. 7)Uncertainty and Robustness

The assumption that the posited relationship f is true is strong. Thiscan be relaxed, by allowing f to vary over an uncertain range.Variations can be introduced manually, for example by stress-testingspecific parameters or functional relationships in f; or automatically,for example, by Monte-Carlo sampling of functions f_(k); k=1, . . . , K,where K is the number of Monte-Carlo draws. For this purpose, we setf_(k)=f(X,A;β_(k)), where the β_(k) are random realizations of modelparameters, which are drawn from a distribution located around the mostlikely values β. So the functions f_(k) are located around the mostlikely function, f. Expected profit becomes a random variable. Therandom profits with and without the innovation, for the k'th Monte-Carlodraw, are:Eprofit_(k) ⁺(X)=f (X,A ⁺;β_(k))  (Eq. 8)Eprofit_(k) ⁻(X)=f (X,A ⁻;β_(k))  (Eq. 9)

The associated random totals and means are:

$\begin{matrix}{{{Eprofit}_{k,\;{total}}^{+} = {\sum\limits_{i = 1}^{N}{{Eprofit}_{k}^{+}\left( X_{i} \right)}}}{{Eprofit}_{k,\;{mean}}^{+} = {\frac{1}{N}{Eprofit}_{k,\;{total}}^{+}}}} & \left( {{Eq}.\mspace{14mu} 10} \right)\end{matrix}$

(Analogous for Eprofit⁻). The random innovation opportunities are:Oppt_(k,total)=Eprofit_(k,total) ⁺−Eprofit_(k,total) ⁻Oppt_(k,mean)=Eprofit_(k,mean) ⁺−Eprofit_(k,mean) ⁻  (Eq. 11)

Uncertain distributions for random profits and random opportunities canbe plotted and standard deviations and confidence intervals can bederived. If the opportunity distribution is positive, then the value ofthe innovation is robust under uncertainty about the true relationship.

Sources of Approximation Error

Approximation error arises from two sources: bias, and variance. Anestimated model is biased if the model form is incorrectly specified,for example, if the model misses important variables. This is the casein the first implementation, where the true model depends on thetransaction score while the estimated model f⁻ does not.

The variance of an estimate or a prediction depends on: properties ofthe development sample used to fit the model; where the predictions aremade; and details of the inference technique. In the secondimplementation below, a novel reject inference technique improves on theextrapolation into the rejected population. In the third implementation,experimental design enriches the development sample by collecting richinformation about the true relationship.

AVACS is at the core of each of these implementations. There are,however, differences in the process of arriving at the estimates f⁺, f⁻.Specifics are presented in the sections below devoted to eachImplementation.

As described above, the methodologies comprising the invention areimplemented by means of executable code running on a computing devicethat includes at least a processing core, storage means, input/outputdevices and means for communicatively coupling all elements of thecomputer, such as a data bus. Therefore, in one embodiment, theinvention is a software simulation engine made up of one or more unitsof executable code, which, when executed, perform the various stepsinvolved in the methods herein described and outputting an estimate ofthe value, or opportunity of an analytic innovation. FIG. 3 is a blockdiagram of a software simulation engine 300. As above, the simulationengine 300 includes one or more functional units. In one embodiment ofthe invention, each functional unit may constitute a discrete program,program unit or software object. In another embodiment, the simulationengine 300 may be a single computer program that performs substantiallyall of the functions depicted in FIG. 3. As described above, thesimulation engine 300, accepts as input (1) data concerning consumerbehavior 304, and/or (2) actions to be taken in regard to the consumerbased on the data 302. The data is input to a model 306 as previouslydescribed. The engine 300 further includes:

a component 308 for developing estimates of profit with and without theanalytic innovation;

a component 310 for using estimates to develop estimated optimalstrategies 210;

a component 312 for computing the profits arising from the estimatedstrategies; and

a component 314 for computing the profit difference. The engine outputsan opportunity estimate 316 of the innovation.

It will be appreciated that the foregoing embodiment of the simulationengine 300 is general in nature. Other embodiments of the simulationengine may contain fewer of the components shown in FIG. 3, or differentcomponents than shown in FIG. 3. For example, the invention encompassesa variety of methodologies as described below. Various embodiments ofthe simulation engine include components for performing any and/or allof the described methodologies. An embodiment of the simulation engineis preferably coded using vector programming techniques in order to makeoptimally efficient use of memory and other computing resources.

As shown in FIG. 4, a further embodiment of the invention is anapparatus 400 for simulating an analytic value chain. The apparatusincludes a simulation engine 300 as previously described. The drawingshows the engine 300 as residing in the memory 408 of the apparatus 400.Additionally, the engine may reside at least partially in the processor404 and/or a mass storage device (116, FIG. 1) such as, for example, ahard disk drive. Typically, the engine 300, configured to perform one ormore methodologies for simulating an analytic value chain to yield anestimate of the opportunity provided by an analytic innovation,instructs the processor to perform operations as shown in FIGS. 5-15.Additionally, the simulation engine instructs the processor to acceptinputs as described herein from one or more input devices 402, such as,for example, a keyboard, a key pad, a mouse, or a data feed delivered toa data port through a wired or wireless connection. The operations ofFIGS. 5-15 having been performed on the input, the simulation engine 300instructs the processor 404 to calculate an estimate of the opportunityprovided by analytic information and output the estimate via an outputdevice 406, such as a display, a printer, or a data port configured totransmit said estimate via a wired or wireless connection.

Value of Transaction Risk Score for Credit Card Line Management

Motivation

A transaction-based risk score can be thought of as a regressionfunction that models default probability as a function of transactiontime series features and other features indicative of risk. Standardmeasures of score power such as area under a ROC (receiver operatingcharacteristic) curve, KS (Kolmogorov-Smirnov test), and Divergenceprovide strong evidence that credit card transaction data containsignificant information about the risk of an account or a consumer thatcannot be explained by credit bureau risk scores and behavior scoresalone. Research performed by the Applicant has shown, for example, thata transaction risk score can identify 5% more delinquent accounts at atypical authorization cutoff, compared to a behavior score.

Simulation Setup

FIG. 5 shows a block diagram of a method 500 for estimating value of atransaction risk score for credit card line management.

We posit a model of the form:Eprofit=f (X,CLI;β)for expected future profit over some suitable time frame from a creditcard operation that is controlled by a line increase strategy 502. Here,A includes a transaction risk score as the only transaction-basedvariable in the model, and other key variables that measure accountbehavior (such as behavior risk-score, revolving balance, utilization,etc.). CLI represents potential credit line increase amounts, rangingfrom $0 to $3 k, in steps of $500. The expected profit model is acomposite made up from various nonlinear regression functions andequations for profit drivers, such as default probability, expectedloss, expected balance, expected revenue and attrition probability. Themodel is informed by analyzing pooled data from multiple lenders thatcapture a variety of line increases and identify account behavior beforeand after a line increase 504.

Next, future profit is estimated using models that include and excludetransaction risk score as a data source, respectively 506. In thisimplementation, we focus on the loss of accuracy of the estimation ifthe transaction risk score is excluded as a data source. We set f⁺(X,CLI,β⁺)=f (X,CLI,β), which is our best approximation. We determine f⁻(X,CLI,β⁻) as our best constrained approximation that does not include thetransaction risk score or any other transaction variables. The set ofmodel parameters β⁻ is constrained such that the transaction scoredoesn't contribute to the profit estimate. The estimated optimalstrategy for f⁺ depends on the transaction score but not on the behaviorscore. The estimated optimal strategy for f⁻ depends on the behaviorscore but not on the transaction score.

Simulation Results

The differences in the strategies lead to differences in expected profit508. The results indicate a significant opportunity of using transactionscores for credit line management:

TABLE 1 Eprofit_(mean) ⁻ Eprofit_(mean) ⁺ Oppt_(mean) $44.20 $50.22$6.02

FIG. 6 shows the distributions of uncertain expected profit for the twostrategies. The expected profit in the strategy that omitted

The uncertain future population default rate was the main driver ofuncertainty in this simulation. Strategic use of the transaction scoreshifts the distribution of expected profit to larger values. But thegraph of FIG. 1 does not tell how reliable the opportunity is in thepresence of uncertainty. The graph of FIG. 7 provides the answer byplotting the corresponding distribution of opportunity from thetransaction.

Monte-Carlo simulation results indicate that the transaction scoreconstitutes a valuable and robust source of information for designingcredit line increase strategies.

It should be appreciated that this simulation applies to any data sourceor data variable that serves to improve the approximation f⁺.

(a) Value of a Novel Reject Inference Technology

Motivation

The problem of loan origination, stated in its simplest form, is todecide whether to accept or to reject an applicant based on somecharacteristics X. The standard solution is to develop a score S(X) thatrank-orders by default risk and to accept only those applicants who passa certain cutoff. As part of score development, there arises the problemof inferring the default probabilities of the rejected population from atruncated sample of accepts for whom we know performance. The simplesttechnique, which serves as a natural benchmark, is straightforwardextrapolation from a regression model that is fit on the acceptpopulation, also called a Known Good/Bad model.

A problem with this approach arises if there are lurking variables thatwere used by the historic screen, but are not included in X. Thissituation is sometimes called “cherry picking.” For example, a branchmanager may have used his personal impression of an applicant tooverride a scoring system. If this is the case, domain expertise canoften add value to a reject inference. For the purpose of the invention,however, it is assumed that X includes all the variables used by thehistoric screen, which is a reasonable assumption for many modernscoring operations.

Another problem for straightforward extrapolation arises if the extentof truncation is large. There is then very limited information on whichto base a regression model for the entire population. The further it isdesired to extrapolate into the rejects, the less accurate thepredictions tend to be. Our innovation focuses on improving the accuracyof the extrapolation. It does this by utilizing information about thehistoric screen in the inference process. FIG. 8 provides a blockdiagram of a method for improving accuracy of a screen for predictinggood/bad status of loan applicants based on reject inference.

Simulation Setup

To link the Accept/Reject decision to expected profit, we posit thefollowing relationship 802:

$\begin{matrix}{{Eprofit} = {{f\left( {X,{A;\beta},g,l} \right)} = \left\{ \begin{matrix}{{{p_{G}(X)}g} - {\left( {1 - {p_{G}(X)}} \right)\; l}} & {{{if}\mspace{14mu} A} =^{``}{Accept}^{''}} \\{0\mspace{236mu}} & {{{{if}\mspace{14mu} A} =^{``}{Reject}^{''}}\;}\end{matrix} \right.}} & \left( {{Eq}.\mspace{14mu} 12} \right)\end{matrix}$

-   -   where:

X: Data available about the loan applicant

A:“Accept”/“Reject”

p_(G): Posited probability of applicant being good

g: Constant gain associated with bad loan

l: Constant loss associated with bad loan

This formula is illustrative only. More complex profit formulas areconceivable, for example, by allowing for gain and loss to depend on X.Whatever the form of the profit function, the principal AVACS approachremains unaltered.

p_(G) represents a score on a probability scale, which is represented bya Generalized Additive Model (GAM) of the form:

$\begin{matrix}{{p_{G}\left( {X;\beta} \right)} = \frac{1}{1 + {\exp\left( {{- \beta_{0}} - {\beta_{1}\;{R\left( {{X;\beta_{2}},\ldots\;,\beta_{k}} \right)}}} \right)}}} & \left( {{Eq}.\mspace{14mu} 13} \right)\end{matrix}$

-   -   where:

R: Score from a posited scorecard

β₀,β₁: Parameters for linear log (Odds) to score transformation

β₂, . . . , β_(k): Score weights

In this application we focus on the sampling error in the probabilityestimates p_(G) ⁻ and p_(G) ⁺, 804 obtained by the straightforwardextrapolation method and by our novel reject inference technique,respectively. The corresponding profit estimates, Eprofit⁻ and Eprofit⁺,are obtained by plugging the probability estimates into equation 12(808). A GAM (generalized additive model) is of a class that capturesnonlinear relationships between predictive variables and the score.

To generate these estimates, we start by creating a clarvoyant sample ofloan applicants and associated binary random outcomes (X,Y∈{0,1}) 806.The applicant characteristics are taken from an empirical sample of loanapplicants. The random outcomes are obtained by sampling from aBernoulli distribution with parameter p_(G):Y=Bernoulli(p _(G)(X))  (Eq. 14)

The clairvoyant sample thus arises from the posited relationship, bydesign.

Next, we generate a development sample 806. For this, we truncate the Yby means of a simulated historic selection process. We model thisselection process by positing a historic application scorecard Q(X) anda cutoff:

$\begin{matrix}{Y_{known} = \left\{ \begin{matrix}{{Y\mspace{14mu}{if}\mspace{14mu} Q} \geq {{historic}\mspace{14mu}{cutoff}}} \\{{Missing}\mspace{14mu}{otherwise}}\end{matrix}\; \right.} & \left( {{Eq}.\mspace{14mu} 15} \right)\end{matrix}$

Based on the development sample, we fit a Known Good/Bad scorecard usingconstrained maximum likelihood estimation. Scorecards approximate log(Odds) as a sum of characteristic scores, where the characteristicscores are stair functions in the variables X. The height of the stairsis given by the score weights for the levels of X. In our Known Good/Badscorecard, we actually constrain the stair functions to typicallymonotone or unimodal shapes, which helps to stabilize the extrapolationinto the truncated region. The restrictions are based on experience andtheoretical considerations. Similar to bin smoothing, applying shaperestrictions is a subjective act of model specification. This technique,as described in Introduction to Model Builder Scorecard, Fair IsaacWhite Paper, (2005) results in probability estimates p_(G) ⁻ andassociated profit estimates Eprofit⁻. Our proprietary innovation resultsin probability estimates p_(G) ⁺ and associated profit estimatesEprofit⁺.

The estimated optimal origination strategies are:

$\begin{matrix}{A^{-} = \left\{ {{\begin{matrix}{{\;^{``}{Accept}^{''}\mspace{14mu}{if}\mspace{14mu}{Eprofit}^{-}} > 0} \\{\;^{``}{Reject}^{''}\mspace{14mu}{otherwise}}\end{matrix}A^{+}} = \left\{ \begin{matrix}{{\;^{``}{Accept}^{''}\mspace{14mu}{if}\mspace{14mu}{Eprofit}^{+}} > 0} \\{\;^{``}{Reject}^{''}\mspace{14mu}{otherwise}}\end{matrix} \right.} \right.} & {{Eq}.\mspace{14mu} 16}\end{matrix}$

Since the p_(G) ⁺ differ from the p_(G) ⁻, differences in the strategiesarise.

It should be appreciated that this simulation applies not only to ourproprietary innovation but to any reject inference innovation thatimproves the estimates p_(G) ⁻.

To define a theoretical benchmark for comparison purposes, the optimalorigination strategy is:

$\begin{matrix}{A^{*} = \left\{ \begin{matrix}{{\;^{``}{Accept}^{''}\mspace{14mu}{if}\mspace{14mu}{Eprofit}} > 0} \\{\;^{``}{Reject}^{''}\mspace{14mu}{otherwise}}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 17} \right)\end{matrix}$

Associated with the optimal strategy is a Hypothetical Optimal Profit,which is attainable if the posited relationship is perfectly well knownto the decision maker.

Simulation Results

We were interested under which operating conditions the novel rejectinference technique would result in an opportunity. The principalparameters for our investigation were the extent of extrapolation(governed by the historic cutoff in relation to the estimated breakevenodds), and the degree of rationality of the historic screen (governed bythe differences between the historic application score Q and the positedscore R). The choice or these parameters generated a number of relevantscenarios: (i) mildly irrational historic screen with portfolioexpansion, (ii) mildly irrational historic screen with portfoliocontraction, and (iii) very irrational historic screen. Since samplingvariation could lead to a “winner” by chance, we generated manyclairvoyant and development samples from the Bernoulli process, for eachscenario. For each sample, we performed the inferences, estimated theoptimal origination strategies, and calculated the expected profits fromthese strategies, thus generating sampling distributions of expectedprofits.

For scenarios (i) and (ii), above we generated the application scorecardQ1 as a slight modification of the posited scorecard R. This and a moreirrational screen are schematically illustrated in FIG. 9. FIG. 9 showsScore weight patterns for the posited profit model, mildly irrational,and very irrational screen.

For example, the variable CB score does mildly and monotonically affectR, but has more influence in Q1. For scenario (iii), we assumed a moreirrational screen Q2, by altering larger parts of the relationship.

The graph in FIG. 10, smoothed histograms of mean account profit forscenario (i), shows the sampling distributions of mean account profitfor scenario (i), where the historic acceptance rate was 46% and theestimated optimal acceptance rate was 50%.

The profit distribution for the innovation is shifted towards highervalues and exhibits lesser variance, as compared to the straightforwardextrapolation method. Hypothetical Optimal Profit is shown as abenchmark,

Although neither of the two inference techniques were capable ofachieving the Hypothetical Optimal Profit benchmark, this was largelybecause or sampling error. The number of known Goods in the developmentsample far outnumbered the known Bads (700 to 800), so that the limitednumber of Bads drives the sampling variation in this scenario.

For scenario (ii): portfolio contraction, and scenario (iii): veryirrational historic screen, the novel technique performed comparable tostraightforward extrapolation.

We conclude that the new technique appears to be beneficial under atleast the following conditions:

-   -   The variables used in the historic screen are known and used in        score development; and    -   The historic screen was developed in a somewhat rational way;        and/or    -   Portfolio expansion is envisioned.

The benefits over and above the straightforward extrapolation methodarise from the feature of the new technique of reducing variance of theextrapolation into the rejects. The benefits gracefully degrade to zeroif either the historic screen is very irrational, or portfoliocontraction is envisioned.

Use of Experimental Design for Learning Strategies in a ChangingEnvironment

Motivation

Learning and adaptation of decision strategies over time as economicconditions, competitors, portfolio composition, and consumer behaviorchange is crucial for the success of a business operation. In thiscontext, it is useful to think about the analytic value chain as afeedback loop, shown below, where data, models, and strategies areconstantly evolving with the goal of increasing profit or maintaining itat a high level.

Champion/Challenger testing is a testing process that compares theeffectiveness of an existing strategy—the champion strategy—with that ofan alternative strategy—the challenger strategy in order to identify themore successful strategy. Adaptive control with champion/challengertesting is an important embodiment of such a feedback system, where thechampion strategy is in place, and challenger strategies are contrivedto compete against the champion. Accounts are randomly assigned tochampion and challengers, so that the strategies compete on a levelplaying field. Profit, or any other measure of decision yield, ismeasured over some time period, and the strategy producing the bestbusiness result becomes the new champion for the next round of testing.

The tools of profit modeling and strategy optimization offer a differentlearning paradigm. Profit is estimated as a function of the data and theactions, and the profit-maximizing actions determine the new strategy.This, however, begs the question of how testing can improve profitestimates, and ultimately, profit. Within the context of the invention,we coined the expression “Learning Strategy,” which is used to denote astochastic strategy, as shown in FIG. 11, which emits experimentalactions according to some experimental design.

Simulation Setup

FIG. 13 shows a block diagram of a method for simulating future outcomesof a credit line increase strategy.

We posit a time-dependent relationship 1302 of the form:Eprofit=f (X,CLI;β₁)   (Eq. 18)

for expected profit from a credit card operation that is controlled by aline increase strategy. We assume discrete learning cycles. Therelationship stays constant over a cycle and can jump between cycles. Wefocus on the sampling error in the profit estimates and how this affectsprofit 10.

The simulation is iterative. It is jump-started with an empirical sampleX. 1304 Associated actions CLI are then simulated according to a positedlearning strategy, such as in FIG. 12, showing a simple example oflearning strategy.

Leaves of a decision tree define segments I-IV, which are associatedwith recommended line increases and test designs for alternativeincrease levels. For an account failing into a segment, a random lineincrease is assigned to one of the amounts specified for this segment,according to a multinomial distribution. Preferably, the tests coversafe and potentially profitable ranges and are typically located aroundthe recommended amount.

Mean profit per account from this learning strategy is then calculated1306, using the initial profit model, which is parameterized by β₀.Next, we simulate future outcomes 1308, also based on the parameters β₀.Certain parameters for error distributions are required for thissimulation, which are also posited. This generates our initial strategydevelopment data set. The simulation allows for the posited profit modelto vary over time, as indicated by a time-variant parameter vectorβ_(i). Over learning cycles t=1, . . . , T;

-   -   Estimate profit model based on previous strategy development        data set 1310;    -   Estimate optimal strategy (resulting in recommended line        increases for learning strategy) 1312;    -   Stochastically assign test actions according to test design        1314;    -   Calculate mean profit per account for the learning strategy from        posited model β_(t) 1316; and    -   Update development data set by simulating future account        behavior from β_(t) 1318;    -   where t←t+1.        It is to be appreciated that this simulation applies to any        experimental design that serves to improve the estimation of a        profit model from data collected during the previous learning        cycle        Simulation Results

We simulated the dynamic evolution of data, models, strategies andprofit over 20 learning cycles. Estimation of the profit model wascompletely automated. Thus, apart from specifying the initial modelstructure and constraints, no analyst intervention took place. Theinitial strategy at time t=0 was chosen to be very simple andsub-optimal, by assigning CLI=$1000 to every account, independent of X.This resulted in an initial mean profit of approximately $50. Varioussimplifications were made to reduce the complexity of this simulation.Accordingly, the quoted profit figures are only of exemplary nature.

We perform the dynamic simulation for two learning strategies thatdiffer in their aggressiveness of testing. The conservative learningstrategy tests in a very narrow range around the recommended increase,and the probabilities that tests are taken are small, resulting in asmall number of tests overall. The aggressive learning strategy performsmore aggressive testing, both in terms of test ranges and testfrequencies, resulting in a larger number of tests overall. We chose theposited profit model to remain constant over several learning cycles. Att=10, we change the posited model, and leave it constant thereafter:β₀=β₁= . . . =β₉≠β₁₀= . . . =β₂₀.  (Eq. 19)

The changes concern parameters that describe the reaction of consumersto line increases: we reduced the effect of line increases on futureaccount balance for certain portfolio segments. Such a change could betriggered by a competitor who targets these segments with higher lineoffers, or by an economic shock. The graph of FIG. 15, illustratingprofit over time from two learning strategies shows the time evolutionof profit. The aggressive learning strategy outperforms the conservativelearning strategy after completing the first learning cycle.

While the aggressive learning strategy rapidly delivers high profit, theconservative learning strategy has difficulties in identifying optimaloperating conditions and profit remains sub-optimal. After the learningcycle, the aggressive learning strategy recovers quickly to achieve asomewhat smaller profit, while the conservative learning strategy fallsinto a substantially oscillatory behavior, again failing to identifybetter operating conditions and loosing out against the aggressivelearning strategy. In addition to these two strategies, we also designedan ultra-aggressive learning strategy (not shown). Its profit was lowerthan that of the aggressive learning strategy.

The simulation not only demonstrates the benefit of learning strategies,but aids in their design. If testing is too limited, the estimatedprofit model remains inaccurate and the estimated optimal strategy canbe mislead, thus reducing profit. Adequate testing will generateinformation-rich data, leading to more accurate estimates of the profitmodel. This, in turn, leads to good estimated optimal strategies thatachieve high profit. On the other hand, if testing is too aggressive theimmediate opportunity cost of testing (instead of taking the recommendedactions) out-weighs the future benefit of learning.

As indicated above, there exist alternate methods for estimating thebenefit of testing by simulating outcomes. For example, an alternatemethod for estimating the benefit of testing by simulating outcomes froman experimental design is described infra.

I. Methodology to Estimate Benefit of Updates to Component Models(assumes the presence of an estimate of updated model as described instage II below).

Let us use the notation Ψ (D_(O) ^(*), f₀(D, X), X) to mean the value ofthe objective function on the data set X at the optimal configurationD_(O) ^(*) of decisions/actions made. The superscripts indicate that wehave an optimal, while the subscript indicates that it is the optimalobtained for a particular version of the component models, f₀(D,X). Thegoal of the present methodology is the improvement of these componentmodels. We want to estimate the beneficial effect of any such update onΨ. As shown in FIG. 14A, in overview, the present methodology 1400includes at least the following steps:

-   -   Perform optimization with old/present suite of models: store        D_(O) ^(*) for further use 1402;    -   Perform optimization with new/estimated model suite f₀(D,X).        Call the result Ψ(D_(n) ^(*), f_(n) (D, X),X) 1404;    -   Evaluate profitability of old solution D_(O) ^(*) with        new/estimated model suite f_(n)(D,X). Call result Ψ (D_(O) ^(*),        f_(n)(D, X), X) 1406:    -   It is preferable to check that solution D_(O) ^(*) is still a        feasible solution. If not, it is preferable to pare back an        offer assignment using another optimization run wherein the        assignment fractions D_(O) ^(*) now serve as new constrained        upper bounds 1408; and    -   Estimated benefit=Ψ(D_(n) ^(*, f) _(n)(D, X), X)−Ψ(D_(O) ^(*, f)        _(n)(D, X), X) 1410.

The sensitivity of the objective function to component models could beestimated by rerunning the optimization in “I” with the output of thecomponent model as a global constraint, so that there is no change inthe solution generated. However, a dual solution also produces thedesired sensitivity analysis.

II Model Estimation and Benefit Simulation Methodology 1400:

-   -   i. assume a full set of exemplars used to train and validate        “old” model;    -   ii. Using some experimental design technique, D-optimality for        example, select new data points 1414;    -   iii. Use old model to predict value of decision targets for the        new data points. In one embodiment of the invention, the        decision targets are

customer value tags, which quantify the value of a customer to abusiness entity 1416.

Typically, the value of the customer to the business entity is anestimate of the potential profit to be made from the customer. Thebusiness entity bases decisions relating to the customer, such aswhether or not to increase the customer's credit line, at least in part,on the value tags. Thus, the value tags, serve as a tool for balancingrisk and potential profit of a particular business decision relating toa customer.

The error-distribution of the old model must be used in this generation.Ideally the error distribution will be Gaussian, with the center equalto output of old model, and width equal to the error of the old model onthe historical validation set. However, this is likely to beheteroskadastic and, thus, the center may require an offset in certainregions of phase space;

-   -   iv. Rebuild the model with the new target 1418;    -   v. Obtain the expected benefit using Methodology described in        section (I) 1420;    -   vi. Iteratively repeat steps iii-v. It should be appreciated        that each iteration in general involves a new set of targets        generated from the expected error distribution of the old model        1422;    -   vii. Total benefit=expectation (i.e. average) of the iterations        in (vi) 1424.

The generation of tags is clear when testing only a single customersegment. Tests across multiple customer population segments may need toinclude correlations. The naive alternative is to simply generate tagsfor various customer population segments as though they are independent.An example of when to include correlations would be, for example, if theratio of the old model outputs for two segments is more accurate thanthe value of the model for either segment.

While we articulated the opportunity of closing the feedback loop withlearning strategies for a greatly simplified credit line managementscenario, the range of business problems that could benefit fromlearning strategies is much larger, for example offer testing for creditcard marketing.

Analytic Value Chain Simulation (AVACS) can pinpoint the commercialpotential of analytic innovations and lead to a better understanding ofthe operating conditions under which they add value. This provides animportant input for business investment decisions into new analytics,e.g. whether to invest in transaction scoring. We presented the theoryof AVACS and applied it to estimate the value of transaction scoring formanaging credit lines, to understand the value of a novel rejectinference technique, and to articulate the value of learning strategies,where AVACS also can aid the design of experiments.

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. It will, however,be evident that various modifications and changes may be made theretowithout departing from the broader spirit and scope of the invention asset forth in the appended claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

1. A computer-implemented method for simulating an analytic value chain,the method being implemented by one or more data processors andcomprising: providing, by at least one data processor, first and secondmodels for estimating future profit, said first model including ananalytic absent from said second model; developing, by at least one dataprocessor, estimates of future profit based on said first and secondmodels respectively; based on said first and second models and saidestimates of future profit, iteratively optimizing, by at least one dataprocessor, decision strategies associated with said models to producefirst and second optimized decision strategies, the first optimizeddecision strategy being based on the first model and the estimate offuture profit for the first model, the second optimized decisionstrategy being based on the second model and the estimate of futureprofit for the second model, the decision strategies each encompassing aset of rules that a business operation uses to determine what action totake in a particular circumstance to achieve a desired result;comparing, by at least one data processor, estimates of future profitbased on said first and second optimized decision strategies; andoutputting, by at least one data processor, an indicator of value forsaid analytic based on said comparison; wherein the optimizing comprisesvarying, by at least one of the data processors, said first model bypermitting variation in f, wherein variation is either introducedautomatically by Monte-Carlo sampling of functions f_(k); k=1, . . . ,K, where K is the number of Monte-Carlo draws; wherein the first modelis a model for future profit of the form: Eprofit=f (X,A/β), where X:data available about a consumer at time of decision; A: potentialactions applicable to the consumer; β: model parameters.
 2. The methodof claim 1, wherein the models utilize data comprising any of:application and survey information; transaction patterns and scores; andactions that were previously applied to the consumer or account.
 3. Themethod of claim 1, wherein the potential actions comprise any of:Accept/Reject for account origination; and discrete levels of LineIncrease Amount for credit card line management.
 4. The method of claim1, wherein a form of the profit model and values of model parameters areinformed by data and domain knowledge.
 5. The method of claim 1, whereinat least one of the a decision strategies comprises mapping from accountinformation to an action space, wherein said first optimal strategycomprises: A⁺(X)=arg max_(AeActionSet)f⁺(X,A;β⁺); and wherein saidsecond optimal strategy comprises: A⁺(X)=argmax_(AeActionSet)f⁻(X,A;β⁻).
 6. The method of claim 5, furthercomprising: calculating, by at least one of the data processors, theexpected profit from the two decision strategies according to:Eprofit⁺(X)=f (X,A⁺;β) for the first decision strategy; andEprofit⁻(X)=f (X,A⁻;β) for the second decision strategy.
 7. The methodof claim 6 wherein said indicator of value for said analytic comprisesand estimate of innovation opportunity, and wherein an estimate of totaland mean opportunity respectively comprise: Oppt_(total)=Eprofit_(total)⁺−Eprofit_(total)−Oppt_(mean)=Eprofit_(mean) ⁺−Eprofit_(mean)−.
 8. Themethod of claim 1, further comprising: varying, by at least one of thedata processors, said first model by permitting variation in f bymanually stress testing certain parameters.
 9. The method of claim 1,wherein simulating an analytic value chain comprises any of: estimating,by at least one of the data processors, the value of transaction-basedvariable for credit line management; determining, by at least one of thedata processors, value of a reject inference technique in loanorigination; and simulating, by at least one of the data processors,future outcomes of a credit line increase decision strategy using alearning strategy.
 10. An apparatus for simulating an analytic valuechain comprising: a computing device, said computing device comprising aprocessing component and a storage component; and a simulation engineresiding in said storage component and comprising instructionsexecutable by said processing component, said simulation engineinstructing said processor to perform operations comprising: providingfirst and second models for estimating future profit, said first modelincluding an analytic absent from said second model; developingestimates of future profit based on said first and second modelsrespectively; based on said first and second models and said estimatesof future profit, iteratively optimizing decision strategies associatedwith said models to produce first and second optimized decisionstrategies, the first optimized decision strategy being based on thefirst model and the estimate of future profit for the first model, thesecond optimized decision strategy being based on the second model andthe estimate of future profit for the second model, the decisionstrategies each encompassing a set of rules that a business operationuses to determine what action to take in a particular circumstance toachieve a desired result; comparing estimates of future profit based onsaid first and second optimized decision strategies; and outputting anindicator of value for said analytic based on said comparison; whereinthe optimizing comprises varying said first model by permittingvariation in f, wherein variation is either introduced automatically byMonte-Carlo sampling of functions f_(k); k=1, . . . , K, where K is thenumber of Monte-Carlo draws; wherein the first model is a model forfuture profit of the form: Eprofit=f (X,A/β), where X: data availableabout a consumer at time of decision; A: potential actions applicable tothe consumer; β:model parameters.
 11. The apparatus of claim 10, whereinthe data comprise any of: application and survey information;transaction patterns and scores; and actions that were previouslyapplied to the consumer or account.
 12. apparatus of claim 10, whereinthe potential actions comprise any of: Accept/Reject for accountorigination; and discrete levels of Line Increase Amount for credit cardline management.
 13. The apparatus of claim 10, wherein a form of theprofit model and values of model parameters are informed by data anddomain knowledge.
 14. The apparatus of claim 10, wherein at least one ofthe decision strategies comprises mapping from account information to anaction space, wherein said first optimal strategy comprises: A⁺(X)=argmax_(AeActionSet)f (X,A;β⁺); and wherein said second optimal strategycomprises: A⁻(X)=arg max_(AeActionSet)f (X,A;β⁻).
 15. The apparatus ofclaim 14, said operations further comprising: calculating the expectedprofit from the two decision strategies according to: E_(profit) ⁺(X)=f(X,A⁺;β) for the first decision strategy; and E_(profit) ⁻(X)=f (X,A⁻;β)for the second decision strategy.
 16. The apparatus of claim 15, whereinsaid indicator of value for said analytic comprises an estimate ofinnovation opportunity, and wherein estimates of total and meanopportunity respectively comprise: Oppt_(total)=Eprofit_(total)⁺−Eprofit_(total) ⁻Oppt_(mean)=Eprofit_(mean) ⁺−Eprofit_(mean) ⁻. 17.The apparatus of claim 10, said operations further comprising: varyingsaid first model by permitting variation in f by manually stress testingcertain parameters.
 18. The apparatus of claim 10, wherein the operationof simulating an analytic value chain comprises any of the operationsof: estimating the value of transaction-based variable for credit linemanagement; determining value of a reject inference technique in loanorigination; and simulating future outcomes of a credit line increasedecision strategy.
 19. An apparatus for simulating an analytic valuechain comprising: means for providing first and second models forestimating future profit, said first model including an analytic absentfrom said second model; means for developing estimates of future profitbased on said first and second models respectively; means foriteratively optimizing decision strategies associated with said modelsto produce first and second optimized decision strategies based on saidfirst and second models and said estimates of future profit, the firstoptimized decision strategy being based on the first model and theestimate of future profit for the first model, the second optimizeddecision strategy being based on the second model and the estimate offuture profit for the second model, the decision strategies eachencompassing a set of rules that a business operation uses to determinewhat action to take in a particular circumstance to achieve a desiredresult, the means for iteratively optimizing strategies varying saidfirst model by permitting variation in f, wherein variation is eitherintroduced automatically by Monte-Carlo sampling of functions f_(k);k=1, . . . , K, where K is the number of Monte-Carlo draws; means forcomparing estimates of future profit based on said first and secondoptimized decision strategies; and means for outputting an indicator ofvalue for said analytic based on said comparison; wherein the firstmodel is a model for future profit of the form: Eprofit=f (X,A/β), whereX: data available about a consumer at time of decision; A: potentialactions applicable to the consumer; β: model parameters.
 20. An articleof manufacture for simulating an analytic value chain comprising:computer executable instructions permanently stored on computer readablemedia, which, when executed by a computer, causes the computer toperform operation comprising: providing first and second models forestimating future profit, said first model including an analytic absentfrom said second model; developing estimates of future profit based onsaid first and second models respectively; based on said first andsecond models and said estimates of future profit, iterativelyoptimizing decision strategies associated with said models to producefirst and second optimized decision strategies, the first optimizeddecision strategy being based on the first model and the estimate offuture profit for the first model, the second optimized decisionstrategy being based on the second model and the estimate of futureprofit for the second model, the decision strategies each encompassing aset of rules that a business operation uses to determine what action totake in a particular circumstance to achieve a desired result; comparingestimates of future profit based on said first and second optimizeddecision strategies; and outputting an indicator of value for saidanalytic based on said comparison; wherein the optimizing comprisesvarying said first model by permitting variation in f, wherein variationis either introduced automatically by Monte-Carlo sampling of functionsf_(k); k=1, . . . , K, where K is the number of Monte-Carlo draws;wherein the first model is a model for future profit of the form:Eprofit=f (X,A/β), where X: data available about a consumer at time ofdecision; A: potential actions applicable to the consumer; β: modelparameters.
 21. The article of claim 20, wherein the data utilized bythe models comprise any of: application and survey information;transaction patterns and scores; and actions that were previouslyapplied to the consumer or account.
 22. The article of claim 20, whereinthe potential actions comprise any of: Accept/Reject for accountorigination; and discrete levels of Line Increase Amount for credit cardline management.
 23. The article of claim 20, wherein a form of theprofit model and values of model parameters are informed by data anddomain knowledge.
 24. The article of claim 20, wherein at least one ofthe decision strategies comprises mapping from account information to anaction space, wherein said first optimal strategy comprises: A⁺(X)=argmax_(AeActionSet)f⁺(X,A;β⁺); and wherein said second optimal strategycomprises: A⁺(X)=arg max_(AeActionSet)f⁻(X,A;β⁻).
 25. The article ofclaim 24, wherein the operations further comprise: calculating, by atleast one of the data processors, the expected profit from the twodecision strategies according to: Eprofit⁺(X)=f(X,A⁺;β) for the firstdecision strategy; and Eprofit⁻(X)=f(X,A⁻;β) for the second decisionstrategy.
 26. The article of claim 25, wherein said indicator of valuefor said analytic comprises and estimate of innovation opportunity, andwherein an estimate of total and mean opportunity respectively comprise:Oppt_(total)=Eprofit_(total)⁺−Eprofit_(total)−Oppt_(mean)=Eprofit_(mean) ⁺−Eprofit_(mean)−.
 27. Thearticle of claim 20, wherein the operations further comprise: varying,by at least one of the data processors, said first model by permittingvariation in f by manually stress testing certain parameters.
 28. Thearticle of claim 20, wherein simulating an analytic value chaincomprises any of: estimating the value of transaction-based variable forcredit line management; determining value of a reject inferencetechnique in loan origination; and simulating future outcomes of acredit line increase decision strategy using a learning strategy. 29.The apparatus of claim 19, wherein the data comprise any of: applicationand survey information; transaction patterns and scores; and actionsthat were previously applied to the consumer or account.
 30. Theapparatus of claim 19, wherein the potential actions comprise any of:Accept/Reject for account origination; and discrete levels of LineIncrease Amount for credit card line management.
 31. The apparatus ofclaim 19, wherein a form of the profit model and values of modelparameters are informed by data and domain knowledge.
 32. The apparatusof claim 19, wherein at least one of the decision strategies comprisesmapping from account information to an action space, wherein said firstoptimal strategy comprises: A⁺(X)=arg max_(AeActionSet)f⁺(X,A;β⁺); andwherein said second optimal strategy comprises: A⁺(X)=argmax_(AeActionSet)f⁻(X,A;β⁻).
 33. The apparatus of claim 19, furthercomprising means for calculating the expected profit from the twodecision strategies according to: Eprofit⁺(X)=f(X,A⁺;β) for the firstdecision strategy; and Eprofit⁻(X)=f(X,A⁻;β) for the second decisionstrategy.
 34. The apparatus of claim 33, wherein said indicator of valuefor said analytic comprises and estimate of innovation opportunity, andwherein an estimate of total and mean opportunity respectively comprise:Oppt_(total)=Eprofit_(total)⁺−Eprofit_(total)−Oppt_(mean)=Eprofit_(mean) ⁺−Eprofit_(mean)−.
 35. Theapparatus of claim 19 further comprising: means for varying said firstmodel by permitting variation in f by manually stress testing certainparameters.
 36. The apparatus of claim 19, wherein simulating ananalytic value chain comprises any of: estimating the value oftransaction-based variable for credit line management; determining valueof a reject inference technique in loan origination; and simulatingfuture outcomes of a credit line increase decision strategy using alearning strategy.